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Abstract 

Listening, watching, reading and writing simultaneously in a foreign language is very 
complex. This paper is part of wider research which explores the use of audiovisual 
comprehension questions imprinted in the video image in the form of subtitles and 
synchronized with the relevant fragments for the purpose of language learning and 
testing. Compared to viewings where the comprehension activity is available only on 
paper, this innovative methodology may provide some benefits. Among them, it could 
reduce the conflict in visual attention between watching the video and completing the 
task, by spatially and temporally approximating the questions and the relevant 
fragments. The technique is seen as especially beneficial for students with a low 
proficiency language level. 

The main objectives of this study were to investigate if embedded questions had an 
impact on SFL students' audiovisual comprehension test performance and to find out 
what examinees thought about them. A multimethod design (Morse, 2003) involving the 
sequential collection of three quantitative datasets was employed. A total of 41 learners 
of Spanish as a foreign language (SFL) participated in the study (22 in the control group 
and 19 in the experimental one). Informants were selected by non-probabilistic 
sampling. The results showed that imprinted questions did not have any effect on test 
performance. Test-takers' attitudes towards this methodology were positive. Globally, 
students in the experimental group agreed that the embedded questions helped them to 
complete the tasks. Furthermore, most of them were in favour of having the questions 
imprinted in the video in the audiovisual comprehension test of the final exam. These 
opinions are in line with those obtained in previous studies that looked into experts', 
SFL students' and SFL teachers' views about this methodology (Casan Nunez, 2015a, 
2016a, in press-b). On the whole, these studies suggest that this technique has 
potential benefits for FL learning and testing. Finally, the limitations of the study are 
discussed and some directions for future research are proposed. 

Keywords: Audiovisual comprehension, listening comprehension, multimethod design, 
Spanish as a foreign language, subtitles, video listening test. 


1. Background 

Contrary to speaking and writing, which have observable products, listening 
comprehension occurs in an internal way, invisible to the eyes of the speaker. Owing to 
that, it is complex to study its nature and to arrive at a definitive description. In this 
paper, listening is understood as a process of interpretation of auditory and visual 
information, as suggested by specialists such as Lynch (2012) and Martin Peris 
(1991/2007). Rubin (1995b, p. 7) proposes the following definition: "an active process 
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in which listeners select and interpret information which comes from auditory and visual 
cues in order to define what is going on and what the speakers are trying to express". 
Thus, it is considered that factors such as "proxemics, kinesics and deictics are all part 
of the message. They are not just a sort of gloss on the verbal component" (Riley, 
1979, p. 84). Harris (2003), Lynch (2012) and Riley (1979) have suggested that the 
term listening does not reflect the multimodal nature of most listening comprehension 
situations. From now on, in order to show the dual dimension of this communicative 
activity, the compound listening/audiovisual comprehension will be employed. 

According to a number of authors (Lynch, 2009; Mendelsohn, 1994; Rubin, 1995a, Ur, 
1999), video materials should prevail over audio recordings to practice 
listening/audiovisual comprehension. To begin with, it is consistent with a definition of 
the skill as a process of interpretation of auditory and visual information. Besides, it 
allows the learner to observe the reality of most speaking interactions. In addition to 
that, video has a positive effect on motivation (Flowerdew & Miller, 2005; Ur, 1994, 
1999; Vandergrift & Goh, 2012). Finally, there are arguments in favour of multimodal 
learning. According to the cognitive theory of multimedia learning (CTML), multimedia 
"takes advantage of the full capacity of humans for processing information. When we 
present material only in the verbal mode, we are ignoring the potential contribution of 
our capacity to also process material in the visual mode" (Mayer, 2014, p. 6). Thus, as 
stated by the multimedia principle of the CTML "students learn better from words and 
pictures than from words alone" (Mayer, 2001, p. 63). This theory was not developed 
specifically for learning foreign languages, however, some principles are applicable to 
this field (Plass & Jones, 2005). In relation to the multimedia principle, these authors 
point out that "it is the combination of both visual and verbal presentations of 
information that has most strongly and consistently supported listening and reading 
comprehension and vocabulary acquisition" (p. 479). Video materials should also prevail 
over audio recordings for testing listening/audiovisual comprehension. First, it is 
congruent with the double nature of this communicative activity. Second, it is in 
harmony with common practice in the classroom (Buck, 2001; Gruba, 1997; Pardo- 
Ballester, 2016). Third, it increases the validity of the test (Bejar, Douglas, Jamieson, 
Nissan, & Turner, 2000; Wagner, 2007, 2008, 2010a), its authenticity (Alderson, 2005; 
Bejar et al. 2000; Ockey, 2007; Wagner, 2007, 2008) and its naturalness (Alderson, 
2005). Fourth, "seeing the situation and the participants tends to call up relevant 
schemes" (Buck, 2001, p. 172). Lastly, not using video in language testing may have a 
negative backwash effect. If the skill is tested only using audio recordings, then there is 
a pressure to practice this communicative activity mainly with this sort of materials. 

When designing listening/audiovisual comprehension tasks, it is essential to take into 
account that "it is extremely difficult to listen and write at the same time, particularly in 
a foreign language" (Underwood, 1989, p. 48), and that listening, viewing, reading and 
writing at the same time can be even more difficult. On the one hand, there is a conflict 
of visual attention between viewing a video and completing a written activity at the 
same time. On the other hand, it should be borne in mind that "there is universal 
agreement that working memory when dealing with novel information is very limited in 
capacity" (Sweller, Ayres & Kalyuga, 2011, p. 42). As Vandergrift and Goh (2012) 
highlight, paying attention to the video and the task simultaneously may cause working 
memory overload. This complexity helps to explain the relatively low degree of attention 
paid to the video in studies that researched test-takers' viewing rates during foreign 
language (FL) video-based listening comprehension tests. Ockey (2007) does not supply 
the average watching rate but, from the data, it can be calculated that it is 44.9%. 
Wagner (2007) found that examinees made eye contact with the screen 69% of the 
time, while in a later study (Wagner, 2010b), this figure decreased to 47.9%. These 
authors recorded the participants while taking the tests and they measured the amount 
of time informants look towards the monitor. Suvorov (2015) uses eye-tracking 
technology to investigate how examinees interact with two types of videos: context 
videos and content videos. He discovers that test-takers spend 58% of the time 
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watching content videos and 51% context videos. These low degrees of attention to the 
image may have a negative influence both on understanding the video and on the 
development of the communicative activity. The complexity of the while-viewing phase 
also contributes to explaining the mixed conclusions in (a) studies that compared the 
results obtained by a group after conducting a video-based FL test with those achieved 
by another group that took an audio-only version of the same test, and (b) research 
into test-taker's attitudes towards the image. Some authors do not find significant 
differences in performance between taking a test with video and completing the same 
test with an audio-only version of the audiovisual text (Batty, 2014; Coniam, 2001; 
Gruba, 1993; Londe, 2009), other authors discover that test-takers achieve higher 
scores with the video based test (Sueyoshi & Hardison, 2005; Wagner, 2010a, 2013), 
and Suvorov (2008) finds evidence that results are higher with the audio-only version. 
Similarly, in some studies, examinees have a positive attitude towards the video image 
(Sueyoshi & Hardison, 2005; Wagner, 2010b), and in others they have a negative 
attitude toward the video (Alderson, Clapham & Wall, 1995; Coniam, 2001; Suvorov, 
2008). Of course, other elements may have played a role in these mixed results. These 
include the focus of listening/audiovisual questions, the complexity of the task and the 
text, the way in which the viewing was carried out, the quality of the input, the stress 
generated by the tests, the greater or lesser degree of solidarity between visual and 
verbal information, the type of visuals (content or context visuals), the influence of the 
video cameras on viewing behaviour, and so on. 

In order to keep while-viewing work manageable, it is advisable that written tasks 
involve scarce reading and writing, and that they require few active elements to be 
stored in working memory. It is also recommendable to use the technique of paused 
listening/viewing (see Field, 2008; Stempleski & Tomalin, 2001; Stoller, 1992). Thirdly, 
it is useful to spatially approximate the video and the activity to reduce the time needed 
to shift from one stimulus to another (from the task to the video and vice versa). Lastly, 
as proposed in this study, it may be more beneficial for the learner to see the 
comprehension questions embedded in the video in the form of subtitles and 
synchronized with the relevant fragments (Casan Nunez, 2015a). Basically, 
comprehension questions appear on screen a few seconds before the beginning of the 
fragment to which they are related, they remain visible for the duration of the relevant 
snippet and they disappear when the relevant part of the video finishes. Compared to 
viewings where the activity is available only on paper, this technique could minimize 
conflict in visual attention by spatially and temporally approximating the questions and 
the pertinent scenes, and it could reduce the cognitive strain of the task, since students 
would only need to pay attention to one subtitled question at a time, instead of different 
printed questions on paper. According to Field (2008), unskilled listeners may reach 
working memory overload faster than competent listeners because the former have 
poorly automatized decoding processes and they spend a great deal of working memory 
on decoding. Consequently, the procedure is seen as especially beneficial for students 
with a low proficiency language level. In addition, it can used occasionally in higher 
proficiency levels for two main reasons: firstly, it helps learners keep focused on what 
they are watching compared to viewings with questions on paper; secondly, the study 
reported in Casan Nunez (2016a) shows that learners with a high listening/audiovisual 
comprehension level in SFL (approximately B2+/C1) have positive views on this 
technique. 

This procedure may be employed in different contexts. It is suitable for paper-and- 
pencil listening/audiovisual comprehension tasks for language learning or testing (see 
Figure 1) and for CALL or CALT (see Figures 2 and 3). As for learning and testing 
activities, Alderson et al. (1995, p. 42) state that "the main difference between a test 
and an exercise is that with exercises learners get support: with tests, they do not". A 
video demonstration of this technique is available from https://youtu.be/ALw8XJkrbDQ 
(01/02/2017). 
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5.^De que temas habla e! chico pelirrojo? 


Figure 1. Example of an audiovisual comprehension question embedded in the video in 
the form of a subtitle. "<LDe que temas habla el chico pelirrojo?" [What is the redheaded 
boy talking about?] From the Spanish film Los peores ahos de nuestra vida by Emilio 

Martinez Lazaro. 


Nombre 


Arthur Smith 


Escucha y observa el video En las preguntas de election multiple solo hay una respuesta correcta 



Figure 2. Prototype of an audiovisual comprehension task for CALL. Notice that playback 
controls are available to learners. "<LComo se llama la chica?" [What is the girl's name?]. 
From the Spanish film Opera prima by Fernando Trueba. 
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Nombre 


Milda 


Escucha y observa el video. En las preguntas de eleccidn multiple solo hay una respuesta correcta. 


4 Considerando kxlo el Iragmento. <,que senbmientos puede 
tenor el chico per la chica? Justifica brevemente la respuesta 


2 tDe que habian? 

[—I a) la (amiha y los esturfcos de la chica 
I—| b) la familia y el trabajo de la chica 
I—| b) la casa y el trabaio de la chica 
□ d) la casa y los estudios de la chica 


3 6 Cuai es el tetefono de la chrca? 


1 tCdmo se lama la chica? 
I—| a) Vanesa 
I I b) Violeta 
I I c) Veronica 
l~l d) Virginia 


Puedes tomar notas aqui 


Figure 3. Prototype of an audiovisual comprehension task for CALT. Notice that only the 
play control is available to test-takers. From the Spanish film Opera prima by Fernando 

Trueba. 


Previously, a theoretical framework describing the use of this technique and its potential 
benefits and limitations has been proposed (Casan Nunez, 2015a). The framework was 
developed out of a literature review, the teaching experience with this procedure and 
the comments of a group of experts in teaching Spanish as a Foreign Language (SFL), 
Spanish linguistics and/or the use of technology. Also, small studies have been carried 
out to investigate what SFL university teachers and university students with a high 
listening/audiovisual comprehension level in SFL (approximately B2 +/C1) think about 
this technique (Casan Nunez, 2016a, in press-b). The results suggest that, overall, 
teachers and learners have positive views about this methodology. In addition to 
experts', teachers' and students' views, it is fundamental to find out what effect this 
technique has on learners' audiovisual comprehension and viewing behaviour. 

The main purposes of the current pilot study were to investigate if the technique had an 
impact on SFL students' audiovisual comprehension test performance and to find out 
what examinees thought about imprinted questions. Moreover, it explored some learner 
preferences regarding listening/audiovisual comprehension. The study used datasets 
from previous research that described the development of a listening/audiovisual test 
(Casan Nunez, 2016b, pp. 36-51). The study reported in this paper, however, had 
different objectives; it took into account data that was not inspected and it analysed the 
data addressing the following research questions: 

Research question 1: How important is it for learners to practise listening/audiovisual 
comprehension in the classroom? What type of recordings (audio or video) do students 
prefer for practising listening/audiovisual comprehension in the classroom? Do learners 
think that the visual input helps them to understand what speakers are saying? How do 
students practise listening/audiovisual comprehension outside the classroom? 

Research question 2: Does the use of questions embedded within the video in the form 
of subtitles and synchronized with the relevant fragments in a FL audiovisual 
comprehension test facilitate test-takers' performance? Do the test-takers of the 
experimental group score higher or lower than the test-takers of the control group? 
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It was hypothesized that examinees that take the audiovisual comprehension test with 
questions embedded in the video, that is, students in the experimental group, would 
outperform examinees that took the same test without them, i.e., those in the control 
group. As described in the introduction, imprinted questions could minimize the conflict 
in visual attention between watching the video and completing the task, by spatially and 
temporally approximating the questions and the relevant fragments. Moreover, this 
technique could diminish the cognitive strain of the activity, because learners would 
only need to pay attention to one request each time, instead of several printed 
questions. Nonetheless, a null hypothesis of no difference was tested. 

Research question 3: What are test-takers' attitudes towards the use of questions 
embedded in the video as subtitles? 

• Research question 3.1: To what extent do test-takers in the experimental 
group agree or disagree that embedded questions helped them to complete 
the tasks? 

• Research question 3.2: Is there any subtitled question consistently considered 
more or less helpful than the others by the experimental group? 

• Research question 3.3: Are students in the experimental group in favour of 
having the questions embedded in the video (in addition to having them on 
paper) in the audiovisual comprehension test of the final exam? 

As mentioned in the introduction, the use of subtitled questions has potential benefits. 
In addition, a previous study (Casan Nunez, 2016a) showed that SFL learners had 
positive attitudes towards this methodology. Therefore, in this study, it was 
hypothesized that students in the experimental group would agree that the questions 
embedded in the form of subtitles aided them, and that they would be in favour of 
having them in the test of the final exam. 

2. Method 

2.1. Study design 

A multimethod design (Morse, 2003) was employed. It involved the sequential collection 
of three quantitative datasets that were used basically to answer different subquestions. 
First, participants were surveyed with the purpose of getting to know the sample and 
some of their preferences regarding listening/audiovisual comprehension. Second, an 
audiovisual comprehension test with two variants was administered to find out if there 
were differences in performance between test-takers of the control and experimental 
groups. Third, attitudinal data towards the use of questions embedded in the form of 
subtitles from the experimental group was collected through a questionnaire. 

2.2. Participants 

SFL students were selected by a convenience, non-probabilistic sampling method 
(Dornyei, 2007, pp. 98-99). The instruments were administered in different lessons. 
Some students did not complete the first questionnaire because they missed the lessons 
where they were handed out (see Table 1). 


Table 1. Number of informants that completed each instrument. 


Instruments 

Groups 


Control 

Experimental 

First questionnaire 

18 

18 

Audiovisual comprehension test 

22 

19 

Second questionnaire (only for the experimental 
group) 

n/a 

19 


41 












The EUROCALL Review, Volume 25, No. 1, March 2017 


All participants were enrolled in Spanish II, a foreign language course delivered at the 
Universidade de Coimbra (Portugal). Spanish II had three shifts. Two of them had fewer 
persons registered. Thus, it was decided that the smaller groups completed the same 
version of the test. Randomly, the less numerous shifts were designated control groups 
and the remaining one, the treatment group. All participants had the same Spanish 
teacher and studied Human or Social Sciences degrees. All but two informants were 
between eighteen and twenty-four years old. All were Lusophones except for one 
participant in the experimental group who was of Ukrainian origin. Roughly speaking, 
students had been studying Spanish for a similar amount of time and they had spent an 
analogous amount of time in Spanish-speaking countries. 83.3% of the participants of 
the control group and 88.9% of the informants of the experimental group reported that 
they were very interested or very much interested in cinema. The others answered 
"neutral". That aspect was relevant because the videotexts employed in the audiovisual 
comprehension test were film scenes and a low interest in cinema could have some 
negative influence on performance. A pre-test to determine the informants' level of 
audiovisual comprehension in Spanish was not administered. From the observation of 
the texts and tasks employed for practising listening/audiovisual comprehension in the 
classroom, it was estimated that most students had a B1+ level in this skill (according 
to the Common European Framework for Languages). As described in Casan Nunez 
(2016b, pp. 39-40), in order to estimate the degree of equivalence between the groups 
regarding audiovisual comprehension, participants completed slightly modified versions 
of the first and second tasks from a B1 level listening test designed by Hidalgo de la 
Torre (Coord., 2013: pp. 52-53). Both groups obtained similar high scores. The average 
mark in the control group was 9.566 out of 12 (SD = 1.861) and in the experimental 
one, it was 9.684 out of 12 (SD = 1.827). That suggested that there were few 
differences between the groups and that the B1 level listening tasks were easy for test- 
takers. The Mann-Whitney test for two independent samples confirmed that there were 
no statistically significant differences between the scores of the groups (L/ = 146.5, z = 
-.185, p = .853, r = -.031). 

2.3. Instruments 

To collect the data, three instruments were developed: two questionnaires and an 
audiovisual comprehension test with two versions. The first questionnaire aimed at 
gathering specific information about the sample group: sociodemographic aspects, what 
importance was attributed to the exercise of listening/audivisual comprehension in the 
classroom to learn Spanish, some learning preferences about listening/audiovisual 
comprehension, to what extent video images were considered helpful or unhelpful to 
understand the interlocutors, and how listening/audiovisual comprehension was 
practiced outside the classroom. It included twenty-six items and, following the 
classification of Saris and Gallhofer (2014), it employed open requests and closed 
categorical requests. The development of the instrument involved expert review, a pilot, 
and a study where the repeated-surveys method (Brown, 2001, pp. 171-172) was used 
to calculate its reliability. The questionnaire and a detailed account of its elaboration can 
be found in Casan Nunez (in press-a). 

The audiovisual comprehension test had two variants: a traditional one for the control 
group and an experimental one for the treatment group. An extensive account of the 
planning, design and trialling of both versions of the test can be found in Casan Nunez 
(2016b). In both instances, tasks were available on paper. Besides, in the experimental 
one, questions were embedded in the video in the form of subtitles and were 
synchronized with the relevant fragments (see Figures 4 and 5). 


42 




The EUROCALL Review, Volume 25, No. 1, March 2017 



Figure 4. Screenshots of questions 6 and 7. The photograms belong to Los peores ahos 
de nuestra vida by Emilio Martinez Lazaro. 
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Figure 5. Timeline in Adobe Premiere Pro. Video 1 and Audio 1 tracks correspond to the 
film. Video 2 track shows the timing of questions 6 and 7. 

The target language use domain was the comprehension of informal conversations 
pertaining to the personal domain in Spanish romantic comedies. The construct 
measured three skills: extracting specific information, identifying general ideas and 
recognizing feelings in face-to-face informal conversations. The test was composed of 
two tasks, two texts and seven items. The main features of the texts and the 
experimental items can be seen in Tables 2 and 3. To indicate the film scene, an eight¬ 
digit time code is used. The first three pairs of digits correspond to hours, minutes and 
seconds, respectively, and the last pair, to frames. Thus, 00:02:36:12 designates the 
following point: 2 minutes, 36 seconds and 12 frames of Opera prima. 


Table 2. Main characteristics of the input texts. 


1. Text source 

Videotext (source: CEFR a p. 49) 

Scene from the comedy Opera prima 
by Fernando Trueba where an 
informal conversation takes place. 

Location: from 00:00:00:00 to 

00:02:36:12. 

Videotext 

Scene from the comedy Los peores 
ahos de nuestra vida by Emilio 
Martinez Lazaro where an informal 
conversation takes place. 

Location: from 00:11:10:00 to 
00:12:48:15. 

2. Authenticity 

Genuine 

Genuine 

3. Domain type (source: CEFR* 
page 45) 

Personal 

Personal 

4. Text length 

2 min and 36 s 

1 min and 38 s 
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5. No of participants 

2 

4 

6. Text speed (global impression) 

Normal 

Fast 

7. Accent (all participants) 

Standard 

Standard 

8. How often played 

Once 

Once 

9. Estimated level 

A2/B1 

B2/C1 


*Common European Framework of Reference for Languages: Learning, Teaching and Assessment. 


Table 3. Description of the embedded questions in the scenes from Opera prima (1-4) 
and Los peores ahos de rtuestra vida (4-7). 


Subtitle 

Focus 

Question 

type 

Timing 

No. 

of 

lines 

Font and 
size 

Colour 

Estimated 

level 

1. iComo se llama la 
chica? 

ESI 

Multiple- 

choice 

00:01:10:00 

00:01:37:09 

1 



A1 

2. iDe que hablan? 

IGI 

Multiple- 

choice 

00:01:44:14 

00:02:02:02 

1 



A2/B1 

3. iCual es el telefono 
de la chica? 

ESI 

Open 

question with 
only one 
possible 
answer 

00:02:04:04 

00:02:22:00 

1 

Arial 

Narrow 

36 

White 

(hex 

colour 

code 

#FFFFFF) 

A2 

4. <LQue sentimientos 
puede tener el chico 
por la chica? 

RF 

Open 

00:02:27:08 

00:02:36:12 

2 

A2/B1 

5. <LDe que temas 
habla el chico 
pelirrojo? 

IGI 

Multiple- 

choice 

00:11:40:00 

00:12:25:24 

1 



B2 

6. ZComo se llama la 
chica? 

ESI 

Multiple- 

choice 

00:12:39:04 

00:12:43:24 

1 



A1 

7. <LQue sentimientos 
puede tener el chico 
pelirrojo por la chica? 

RF 

Open 

00:12:45:00 

00:12:48:15 

2 



B2 


Abbreviations: ESI - extracting specific information. IGI - identifying general ideas. RF - recognizing feelings. 


The second questionnaire aimed at answering the third group of research questions. It 
was a development of a survey used in a previous exploratory study (Casan Nunez, 
2016a). A questionnaire design was chosen because they are "uniquely capable of 
gathering large amounts of information quickly" (Dornyei, 2007, p. 101). The 
instrument had three parts: title, introductory text and items (see appendix). The title 
tried to be as informative as possible. The introductory text specified the objective of 
the questionnaire, it mentioned that there were no correct or incorrect answers, and it 
stated that the data collected would be treated confidentially and would only be used for 
academic purposes. Following the classification of Saris and Gallhofer (2014), the 
instrument was made up of two types of items: closed categorical requests and open 
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requests. The former ones were of two subtypes: requests with nominal response 
categories and requests with ordinal response categories. 

As recommended by Saris and Gallhofer (2014), the response options of the closed 
categorical requests attempted to offer a range of options, broad enough to encompass 
the possible answers of participants, and they were mutually exclusive. Closed 
categorical requests with ordinal response categories consisted of Likert-type items (1, 
2.1, 2.2, 2.3, 2.4, 5, 6.1, 6.2 and 6.3). They made up a scale that attempted to answer 
research question 3.1 To what extent do test-takers in the experimental group agree or 
disagree that embedded questions helped them to complete the tasks? The scale was 
named helpfulness scale. Additionally, individual items tried to respond to research 
question 3.2 Is there any subtitled question ranked consistently higher or lower than 
the others by the experimental group? Test-takers expressed their opinion about the 
tasks of the test in a global way (items 1 and 5) and about each subtitled question 
individually (items 2.1, 2.2, 2.3, 2.4, 6.1, 6.2 and 6.3). There are discrepancies on the 
number of anchors of Likert scales: Likert (1932) employs mainly five alternatives; 
Jacoby and Matell (1971) suggest that three categories are sufficient; Allen and Seaman 
(2007) advise a minimum of five options, and Bisquerra and Perez-Escoda (2015) 
recommend eleven. Three anchors were opted for, primarily for two reasons. First, it 
was enough to find out if the attitude towards subtitled questions was positive, negative 
or neutral. Second, the size of the sample was small and it would be necessary to 
collapse the categories. The alternatives chosen were: agree, neither agree nor disagree 
and disagree. In that selection there was a balance between positive and negative 
alternatives and a neutral central point could be identified. In addition, terms that 
indicated total agreement or disagreement were avoided because, according to Cohen, 
Manion and Morrison (2011), people often prefer to skip them so as not to appear 
extremist. Closed categorical requests with nominal response categories (items 3 and 7) 
sought to answer research question 3.3 Are students in the experimental group in 
favour of having the questions embedded in the video (in addition to having them on 
paper) in the audiovisual comprehension test of the final exam? 

"By permitting greater freedom of expression, open-format items can provide a far 
greater richness than fully quantitative data. The open responses... can also lead us to 
identify issues not previously anticipated" (Dornyei, 2007, p. 107). Their main drawback 
lies in the complexity of analysing them (Cohen et al., 2011). Two items of this nature 
were included so that participants could express their opinion freely (items 4 and 8). In 
order for test-takers' production level in Spanish not to restrict responses, the 
possibility of writing both in Spanish and Portuguese was offered. 

2.4. Procedures 

All instruments were administered by the researcher during Spanish II lessons. The fact 
that the author was also the teacher may have favoured the fact that participants took 
the questionnaires and the test seriously (an essential requirement for the results to be 
valid). The first questionnaire was administered for the first time in February 2014 to 
students from the three shifts. Precise instructions were provided and students were 
told that there were no correct or incorrect replies. No time limit was imposed and 
questions arising from any doubts were answered. Informants took up to 8 minutes to 
complete the questionnaire and none of them requested help. 

The test and the second questionnaire were administered in the same classroom in 
March 2014. Treatment groups completed the experimental version of the test and the 
questionnaire, whereas the control group took the traditional variant of the test. Test- 
takers were informed that they could take as long as they needed to answer. The 
slowest test-taker of the control group employed 12 minutes and 27 seconds to finish 
the test. The slowest examinees of the two experimental groups needed 20 minutes and 
14 seconds, and 19 minutes and 42 seconds to respond to the test and the survey. A 
detailed account of the procedures followed can be found in Casan Nunez (2016b, pp. 
42-43). 
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2.5. Data analyses 

All quantitative statistical analyses were carried out using SPSS version 21, except for 
effect sizes. For that purpose, the Windows scientific calculator was used. Data was 
double-checked for accuracy to avoid input errors. Furthermore, a frequency analysis of 
all variables was carried out to verify that there were no missing or anomalous values. 
Table 4 sums up the quantitative data analyses that were followed to answer each 
research question. 

As for the open-ended items in the second questionnaire (4 and 8), only 5 out of 19 
participants in the experimental group made comments, and those observations were 
short (one sentence). This might have to do with Dornyei's cautionary advice (2007, p. 
105): questionnaires "are unlikely to yield the kind of rich and sensitive description of 
events and participant perspectives that qualitative interpretations are grounded in". As 
the amount of qualitative data was so small, the only analysis consisted in quantizing 
related responses. 


Table 4. Quantitative data analyses used for answering research questions 


Research 

question 

Research instrument 

Analyses 

RQ1 

First questionnaire 

Frequencies 


(instrument developed by 
Casan Nunez, in press-a) 


RQ2 

Audiovisual comprehension 

Scores of each version of the test: descriptive statistics, 


test 

analyses of facility, discrimination and reliability, and 


(instrument developed by 
Casan Nunez, 2016b) 

correlations with two tasks from a listening exam (see Casan 
Nunez, 2016b, pp. 44-51) 



Tests scores: Shapiro-Wilk test 



Comparison of tests scores: Mann-Whitney test 

RQ3 

Second questionnaire 

RQ 3.1: Reliability analysis of the helpfulness scale (Cronbach's 


(based on a survey used by 
Casan Nunez, 2016a) 

alpha, corrected item-total correlation, Cronbach's alpha if item 
deleted) and descriptive statistics of the scale 



RQ 3.2: Friedman test (items 2.1, 2.2, 2.3, 2.4, 6.1, 6.2 and 

6.3) and two Wilcoxon Signed Rank tests using a Bonferroni 
adjusted alpha value (item 2.4 with item 2.2, item 2.4 with item 

2.3) . 



RQ 3.3: frequencies (items 3 and 7) 


3. Results and discussion 

3.1. Research question 1 

The first research question addressed learners' thoughts and preferences regarding 
listening/audiovisual comprehension. Informants' answers can be found in Tables 5 to 8. 
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Table 5. How important is it for learners to practise listening/audiovisual comprehension in the classroom? 


Frequency 

Percent 

Valid Percent 

Valid 

Very* 

16 

39.0 

44.4 

Very much 

20 

48.8 

55.6 

Total 

36 

87.8 

100.0 

Missing** 

System 

5 

12.2 


Total 

41 

100.0 



*Participants could choose one of five responses: (a) very little, (b) a little, (c) neutral, (d) very, and (e) very 
much. 

**Missing values correspond to informants that did not complete the first questionnaire. 


All informants considered that practising listening/audiovisual comprehension in the 
classroom was either "very" important (16 / 44.4%) or "very much" important (20 / 
55.6%). These results are congruent with the weight of listening/comprehension in 
communication ("we listen to twice as much language as we speak, four times as much 
as we read, and five times as much as we write", Celce-Murcia & Olshtain, 2000, p. 
102) and in acquisition ("in order for acquirers to progress to the next stage in the 
acquisition of the target language, they need to understand input language that includes 
a structure that is part of the next stage", Krashen & Terrel, 1983, p. 32). 


Table 6. What type of recordings (audio or video) students prefer for practising listening/audiovisual 
comprehension in the classroom? 



Frequency 

Percent 

Valid Percent 

Valid 

Audio* 

4 

9.8 

11.1 

Video 

11 

26.8 

30.6 

Both equally 

21 

51.2 

58.3 

Total 

36 

87.8 

100.0 

Missing** 

System 

5 

12.2 


Total 

41 

100.0 



*Participants could choose one of three options: (a) audio, (b) video, and (c) both equally. 
**Missing values correspond to informants that did not complete the first questionnaire. 


Most students (21 / 58.3%) reported that they did not have a preference for audio or 
video materials for practising listening/audiovisual comprehension in the classroom. In 
addition, many more participants showed a predilection for video (11 / 30.6%) instead 
of audio (4 / 11.1%). As participants were not asked to justify their answer, it is not 
possible to know why they made their choices. Possibly, the preference for video over 
audio could be due to the positive effect of video on motivation (Flowerdew & Miller, 
2005; Ur, 1994, 1999; Vandergrift & Goh, 2012). Despite this, from the point of view of 
the learners' preferences, the results provide support for the idea, previously defended 
in the introduction, that video should prevail over audio to teach listening/audiovisual 
comprehension, or that at least both types of materials should have a similar weight. 
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Table 7. Do learners think that the visual input helps them to understand what speakers are saying? 



Frequency 

Percent 

Valid Percent 

Valid 

A little* 

1 

2.4 

2.8 

Neutral 

9 

22.0 

25.0 

Very 

20 

48.8 

55.6 

Very much 

6 

14.6 

16.7 

Total 

36 

87.8 

100.0 

Missing** 

System 

5 

12.2 


Total 


41 

100.0 



*Participants could choose one of five responses: (a) very little, (b) a little, (c) neutral, (d) very, and (e) very 
much. 

**Missing values correspond to informants that did not complete the first questionnaire. 


Most informants (26 / 72.3%) responded that the visual input was "very" or "very 
much" useful to understand the speakers. One participant commented on item 4 of the 
second questionnaire the following: "Pienso que la imagen es una ayuda para la 
comprension" ("I think the image aids comprehension", author's translation). The 
results are in line with some of the findings in Sueyoshi and Hardison (2005, p. 682) 
and Wagner (2010b, p. 287). Sueyoshi and Hardison asked 42 English as a second 
language (ESL) students if they agreed with three different statements: "it is easier to 
understand English when I can see the speaker's face", "it is easier to understand 
English when I can see the speaker's gestures" and "it is easier to understand English 
conversations on TV than on the radio" (p. 697). Participants' answers revealed 
agreement with the three utterances. 14 of the 42 ESL learners were exposed to a video 
where it was possible to see the speakers' gestures and face. Afterwards, they were 
asked if they agreed that watching the speaker's gestures, on the one hand, and 
watching the speaker's face, on the other hand, helped their understanding. Globally, 
students sympathized with both ideas. Wagner (2010b) administered a post-test 
questionnaire to 56 ESL test-takers to explore their attitudes towards the use of video. 
The survey consisted of seven 5-point Likert-type items. Globally, participants' attitudes 
were positive. One of the items requested informants to express agreement or 
disagreement towards the statement "being able to see the video made the test easier" 
(p. 286) and most informants "agreed". Other authors found evidence that learners had 
a negative view of the video in listening/audiovisual comprehension tests (Alderson et 
al., 1995; Coniam, 2001; Suvorov, 2008). As mentioned in the introduction, those 
mixed results could be due to several reasons (the conflict in visual attention between 
watching the video and completing the task, the focus of the questions, the complexity 
of the task and the text, etc.). 


Table 8. How do students practise listening/audiovisual comprehension outside the classroom? 


Percent 


o* 

1-2 

3-4 

5-6 

7 

Listening to audio recordings intended for language learning 

58.3 

22.2 

16.7 

2.8 

0 

Listening to audio recordings intended for native speakers of 

25 

13.9 

36.1 

25 

0 
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Spanish 






Talking to native speakers in Spanish 

27.8 

41.7 

25 

2.8 

2.8 

Talking to non-native speakers in Spanish 

19.4 

41.7 

30.6 

2.8 

5.6 

Watching videos intended for language learning 

58.3 

16.7 

19.4 

2.8 

2.8 

Watching videos intended for native speakers of Spanish 

25 

27.8 

27.8 

13.9 

5.6 


*0 = they do not, 1-2 = once or twice a week, 3-4 = 3 or 4 times a week, 5-6 = 5 or 6 times a week, 7 = 
everyday 


The most frequent ways to practise listening/audiovisual comprehension outside the 
classroom were listening to audio recordings and watching video materials intended for 
native speakers. The results can be related to a study on audiovisual materials by 
Gonzalez-Vera and Hornero Corisco (2016). The authors asked 37 English as a foreign 
language learners what materials they used to practise listening outside the language 
classroom. Among the five options provided (Podcasts, video clips, DVDs, CDs and 
other), the most chosen categories were video clips (30%), CDs (28%) and DVDs 
(23%). In addition, the results revealed that students preferred authentic materials 
rather than materials intended for language learning. This is in line with the value of 
realia in some current language teaching methods, such as communicative language 
teaching and task-based language teaching. As for the results regarding "talking to 
native speakers in Spanish" and "talking to non-native speakers in Spanish", they seem 
logical because, on the one hand, Spanish as a foreign language was being learnt and, 
on the other hand, in Coimbra there were opportunities to interact with Spanish 
exchange students, Spanish tourists and Spanish residents. 

3.2. Research question 2 

The second research question was Does the use of questions embedded within the video 
in the form of subtitles and synchronized with the relevant fragments in a FL audiovisual 
comprehension test facilitate test-takers' performance? Do the test-takers of the 
experimental group score higher or lower than the test-takers of the control group? 

As described in Casan Nunez (2016b), the scores of both versions of the test were 
subject to descriptive statistics, analyses of facility, discrimination and reliability, and 
correlations with two tasks from a listening test. Overall, the results showed that the 
instruments were adequate for the research purposes for which they were designed. 
Just to mention some aspects, Cronbach's alpha was .650 in the traditional test and 
.705 in the experimental one (according to Suhr & Shay [2009], alphas over .60 are 
acceptable for instruments developed for research purposes); in both tests, the mean 
inter-item correlation for the items was in the .2 to .4 range (as recommend by Briggs & 
Cheek [1986]); and the audiovisual comprehension tests were significantly correlated in 
a positive way with two tasks from a listening exam (experimental test: r 5 = .592; p < 
.01; traditional test: r s = .833, p < .01). 


Table 9. Descriptive Statistics for the tests 


Test 

N 

Min 

Max 

M 

SD 

Mdn 

Skewness 

SE 

Kurtosis 

SE 

Trad. 

22 

1.00 

7.00 

5.182 

1.652 

5.500 

-1.016 

.491 

.731 

.953 

Exp. 

19 

2.00 

7.00 

5.053 

1.779 

5.000 

-.618 

.524 

-.814 

1.014 


Abbreviations: Trad. = traditional. Exp. = Experimental. 
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Table 10. Test Statistics* 



Mann- 
Whitney U 

Wilcoxon W 

Z 

Asymp. Sig. (2- 
tailed) 

Exact Sig. 

(2-tailed) 

Exact Sig. 

(1-tailed) 

Point 

Probability 

Test 

203.500 

393.500 

-.147 

.883 

.885 

.444 

.009 


♦Grouping Variable: type of test (traditional test, experimental test). 


As can been seen in Table 9, the mean scores were very similar in both tests, 
suggesting that questions embedded within the video did not have an effect on test 
performance. Since the distributions of the test scores were not normally distributed 
either on the experimental test (S-tV = .874, df = 19, p = .017) or the traditional test 
(S-W = .888, df = 22, p = .017), it was not appropriate to use an independent-samples 
t- test. Instead, the Mann-Whitney test was employed (see Table 10). The results 
revealed no statistically significant difference between the test scores obtained by the 
control group ( Mdn= 5.500) and the experimental group (Mdn = 5.000), U = 203.500, z 
= -.147, p (exact, two-tailed) = .885, r = -0.023, which indicated that the subtitled 
questions did not have any impact on test performance. The results did not confirm the 
hypothesis that the experimental group would outperform the control group thanks to 
the help provided by the questions embedded in the video. One can think of two 
different explanations for that. First, perhaps the hypothesis was wrong. It could be the 
case that questions embedded in the video as subtitles do not constitute either a help or 
a hindrance when they are combined with questions on paper because learners do not 
pay attention to them. Second, the hypothesis and the reasoning behind it were right 
but they did not hold in the study for some reason. It could be that the groups were not 
equivalent. Although they had a similar profile, participants did not complete a pre-test 
to determine their level of audiovisual comprehension in Spanish. 

3.3. Research question 3 

The third research question investigated test-takers' attitudes towards the use of 
questions embedded in the video as subtitles. Research question 3.1. inquired to what 
extent test-takers in the experimental group agreed or disagreed that embedded 
questions helped them to complete the tasks. A helpfulness scale (items 1, 2.1, 2.2, 
2.3, 2.4, 5, 6.1, 6.2 and 6.3) was employed to try to answer this matter. To measure its 
reliability, Cronbach's alpha, corrected item-total correlation (CITC) and Cronbach's 
alpha if item deleted (CAID) values were calculated. Cronbach's alpha was .778. 
According to Cohen et al. (2011: p. 640), alpha values between .70 and .80 indicate 
that the scale is "reliable". All but one CITC values were above .30 (between .365 and 
.763), as recommend by De Vaus (2002) and Pallant (2011). Item 6.3 had a CITC level 
of .254, which is acceptable for Henning (1987, cited in Green, 2013). CAID values were 
between .709 and .781. In other words, none of the items would substantially increase 
the reliability if they were deleted. As the analysis was satisfactory, a new variable 
containing the sum of the component items was created. For that purpose, each 
"disagreement" was counted as 1 point, each "neither agree nor disagree" as 2 and 
each "agree" as 3. The scale score was checked for outliers and two were identified (see 
Figure 6). However, as Osborne and Overbay (2004, p. 3) point out, "there is a great 
deal of debate as to what to do with identified outliers". Following Larson-Hall's (2010) 
proposal, statistics with and without them were calculated (see Tables 11 and 12). 
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Figure 6. Boxplot of the scale scores. 


Table 11. Descriptive Statistics for the scale with two outliers 


N 

Min 

Max 

M 

Mdn 

SD 

Skewness 

SE 

Kurtosis 

SE 

Scale 

19 

9 

27 

20.42 

21 

4.059 

-1.366 

.524 

2.563 

1.014 

Valid N 

19 



Table 12. Descriptive Statistics for the scale without outliers 


N 

Min 

Max 

M 

Mdn 

SD 

Skewness 

SE 

Kurtosis 

SE 

Scale 

17 

16 

27 

21.47 

22 

2.577 

-.297 

.550 

1.167 

1.063 

Valid N 

17 



The minimum score for the scale could be 9 (for answering "disagree" to all 9 items) 
and the maximum 27 (for responding "agree" to all 9 items). Therefore, the higher the 
score, the greater the agreement that embedded questions helped learners complete 
the test. The mean of the scale with outliers (M = 20.42) suggests that, overall, 
participants considered that imprinted questions were useful. The mean of the scale 
without outliers (A7 = 21.47) revealed the same fact even more clearly. The results 
support the hypothesis and they are in line with those obtained in a previous study with 
SFL learners (Casan Nunez, 2016a). In that investigation the mean of the scale was 
11.50 out of 15 (SD = 2.565). 

Research question 3.2 explored whether any subtitled question was consistently 
considered more or less helpful than the others by the experimental group. The results 
of the Friedman test (see Table 13) indicated that there were statistically significant 
differences in the ranking of the embedded questions, x 2 (6, n = 19) = 16.653, p = 
.011. Two embedded questions had particularly low mean ranks: 4 and 7. They differed 
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from the others in two ways. First, they measured the ability to recognize feelings (see 
table 3) and second, they did not have a focalising character: they implied global 
comprehension and the subtitles appeared at the end of the videos. Wilcoxon Signed 
Rank tests were conducted to follow up the findings. A Bonferroni adjusted alpha value 
was used to reduce the chances of obtaining false-positive results. As recommended by 
Field (2009) and Pallant (2011), selective comparisons were chosen to keep the alpha 
at a manageable level. Two Wilcoxon Signed Rank tests were carried out. Thus, the 
alpha level of significance was .025. Embedded question 4 (focusing on recognizing 
feelings, not focalised in the pertinent scene, level A2/B1) was compared with 
embedded questions 2 (focusing on identifying general ideas, synchronized with the 
relevant fragment, level A2/B1) and 3 (focusing on extracting specific information, 
synchronized with the pertinent fragment, level A2). Wilcoxon tests revealed that 
embedded question 4 was ranked significantly lower than question 2, z = -2.311, p 
(exact, two tailed) = .024, r = -.375, and question 3, z = -2.443, p (exact, two tailed) 
= .018, r = -.396. These facts could be due to the lack of focalization of embedded 
question 4. However, it would be necessary to ask test-takers to disclose how they 
evaluated the helpfulness of the imprinted questions. 


Table 13. Results from the Friedman test. 

Ranks 

Questionnaire item / Subtitled question 

Mean Rank 

2.1. / 1 

3.79 

2.2. / 2 

4.74 

2.3. / 3 

4.68 

2.4. / 4 

3.11 

6.1. / 5 

3.92 

6.2. / 6 

4.55 

6.3. / 7 

3.21 


Test Statistics* 

N 

19 

Chi-Square 

16.653 

df 

6 

Asymp. Sig. 

.011 

*Friedman Test 


Research question 3.3. investigated whether students in the experimental group were in 
favour of having the questions embedded in the video in the audiovisual comprehension 
test of the final exam. In order to address that matter, test-takers were asked directly 
after completing each task of the test (items 3 and 7 of the second questionnaire). The 
answers were virtually the same both times: most informants were in favour and very 
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few against this (see Tables 14 and 15). These results suggest that students considered 
that embedded questions were useful. 


Table 14. Item 3 (task 1). Are you in favour of having the questions embedded in the image (in addition to 
having them on paper) in the audiovisual comprehension test of the final exam? 



Frequency 

Percent 

Valid Percent 

Valid 

No 

2 

10.5 

10.5 

I am not sure 

4 

21.1 

21.1 

Yes 

13 

68.4 

68.4 

Total 

19 

100.0 

100.0 


Table 15. Item 7 (task 2). Are you in favour of having the questions embedded in the image (in addition to 
having them on paper) in the audiovisual comprehension test of the final exam? 



Frequency 

Percent 

Valid Percent 

Valid 

No 

2 

10.5 

10.5 

I am not sure 

3 

15.8 

15.8 

Yes 

14 

73.7 

73.7 

Total 

19 

100.0 

100.0 


4. Conclusion 

Listening, watching, reading and writing at the same time in a foreign language is a 
cognitively demanding task. This paper is part of wider research which investigates the 
use of audiovisual comprehension questions imprinted in video images in the form of 
subtitles and synchronized with the relevant fragments, for the purpose of language 
learning and testing. Compared to viewings where the task is available only on paper, 
this technique may provide some benefits. Among them, it could reduce the conflict in 
visual attention between watching the video and completing the task, by spatially and 
temporally approximating the questions and the relevant scenes. The procedure is 
mainly intended for students with a low proficiency level. 

This pilot multimethod study (Morse, 2003) investigated for the first time whether 
questions embedded in videos as subtitles had any impact on FL learners' audiovisual 
comprehension test performance. The results suggest that they do not have any effect 
on test performance. Test-takers' attitudes towards this technique are positive, 
however. Overall, participants in the experimental group agree that the imprinted 
questions help them to complete the tasks. Furthermore, most of them are in favour of 
having the questions embedded in the video in the audiovisual comprehension test of 
the final exam. Test-takers' opinions can be paralleled to studies that research experts', 
SFL students' and SFL teachers' views about this methodology (Casan Nunez, 2015a, 
2016a, in press-b). Globally, all three groups have positive opinions. Experts agree or 
strongly agree that the technique can be useful for the teaching of listening/audiovisual 
comprehension, and that it can provide various benefits compared to viewings where 
the activity is available only on paper; among them, that it can minimize the conflict in 
visual attention between watching a video and completing a task at the same time, and 
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that it helps FL students to focus their attention towards the viewing objectives. SFL 
teachers think that embedded questions are beneficial for FL learning, and they coincide 
with the experts in some of the advantages. SFL students agreed that imprinted 
questions helped them to complete a testing task; besides, their comments imply that 
subtitles focalise attention in the relevant moments, and that they minimize the conflict 
of visual attention. All in all, although these studies are limited and further empirical 
research is needed, the positive opinions suggest that this technique has potential 
benefits for FL learning and testing. 

The current study has a number of shortcomings. First, SFL students were not selected 
by probabilistic sampling. As Dornyei (2007) points out, this weakness is present in 
most experimental studies in the social sciences. Second, treatment diffusion 
constituted a potential threat to internal validity because the groups took the 
audiovisual test consecutively in the same classroom. Two circumstances reduced the 
possibility of the exchange of information: (a) there was little time for it to happen, 
since test-takers took the tests consecutively and they had different class schedules; 
and (b) in principle, there was no obvious interest in knowing the test content, as it had 
no impact on the final grade. A further limitation consists in the impossibility of 
guaranteeing that the groups were of equal ability. Although they were similar in many 
aspects and there were no statistically significant differences between the groups' 
performance on two tasks from a listening test, an audiovisual comprehension test to 
find out the participants' level of audiovisual comprehension in SFL was not 
administered. 

This leads us to some directions for future research. First, it is important to investigate 
whether embedded questions have an impact on FL learners' viewing behaviour with 
regard to the video image. Compared to viewings where the task is available only on 
paper, imprinted questions may reduce the conflict in visual attention between watching 
the video and completing the task, by spatially and temporally approximating the 
questions and the relevant fragments. Based on this, it is hypothesized that embedded 
questions may increase the amount of time devoted to viewing the video by the learner. 
Second, it would be beneficial to replicate this study with a larger sample and compare 
the results. Third, it would be useful to carry out similar studies with tests for different 
target language use domains, other text types and other text lengths. Lastly, as this 
work explored embedded questions in a testing situation, further research is needed to 
investigate this technique in a learning context. 


Dedication 

This article is dedicated to my father. 
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Appendix 

Preguntas integradas como subtitulos: la opinion del estudiante (original 

version) 

El presente cuestionario tiene por finalidad conocer que piensas sobre el uso de las 
preguntas de comprension integradas en el video en forma de subtitulos. No existen 
respuestas correctas ni incorrectas porque estas expresando tu opinion. La informacion 
que proporciones se tratara de forma confidencial y solo se utilizara con fines 
academicos. 

Tarea 1. Fragmento de la pelicula Opera prima. 

1. Sobre la base de tu experiencia global completando la tarea, <len que medida estas 
de acuerdo o en desacuerdo con que las preguntas subtituladas te han ayudado? Senala 
la respuesta con un circulo. 

a) En desacuerdo 

b) Ni de acuerdo ni en desacuerdo 


c) De acuerdo 

2. De forma individual, den que medida estas de acuerdo o en desacuerdo con que las 
preguntas subtituladas te han ayudado? Completa la tabla poniendo equis (X). 



En desacuerdo 

Ni de acuerdo ni en 
desacuerdo 

De acuerdo 

2.1. dComo se llama la chica? 




2.2. dDe que hablan? 




2.3. dCual es el telefono de la chica? 




2.4. iQue sentimientos puede tener el 
chico por la chica? 





3. dEstas a favor de que en la prueba de comprension audiovisual las preguntas 
aparezcan en la imagen (ademas de en papel)? 


a) No 

b) No estoy seguro 

c) Si 

4. A continuacion tienes un espacio en el que puedes comentar cualquier aspecto de la 
actividad. Puedes responder en espanol o portugues. 

Tarea 2. Fragmento de la pelicula Los peores ahos de nuestra vida 

5. Sobre la base de tu experiencia global completando la tarea, den que medida estas 
de acuerdo o en desacuerdo con que las preguntas subtituladas te han ayudado? Senala 
la respuesta con un circulo. 
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a) En desacuerdo 

b) Ni de acuerdo ni en desacuerdo 


c) De acuerdo 

6. De forma individual, <Len que medida estas de acuerdo o en desacuerdo con que las 
preguntas subtituladas te han ayudado? Completa la tabla poniendo equis (X). 



En desacuerdo 

Ni de acuerdo ni en 
desacuerdo 

De acuerdo 

6.1. iDe que temas habla el chico pelirrojo? 




6.2. iComo se llama la chica? 




6.3. Considerando todo el fragmento, ique 
sentimientos puede tener el chico pelirrojo 
por la chica? Justifica brevemente la 
respuesta. 





7. <LEstas a favor de que en la prueba de comprension audiovisual las preguntas 
aparezcan en la imagen (ademas de en papel)? 

a) No 

b) No estoy seguro 

c) Si 

8. A continuacion tienes un espacio en el que puedes comentar cualquier aspecto de la 
actividad. Puedes responder en espanol o portugues. 


Questions embedded as subtitles: students' opinion (English translation) 

This questionnaire aims to know what you think about the use of audiovisual 
comprehension questions embedded in the video in the form of subtitles. There are no 
right or wrong answers because you are expressing your opinion. The information you 
provide will be treated confidentially and will only be used for academic purposes. 

Task 1. Fragment from the film Opera prima. 

1. Based on your overall experience completing the task, to what extent do you agree 
or disagree that the subtitled questions have helped you? Mark your answer with a 
circle. 


a) Disagree 

b) Neither agree nor disagree 


c) Agree 

2. Individually, to what extent do you agree or disagree that subtitled questions have 
helped you? Complete the table with "X". 



Disagree 

Neither agree nor disagree 

Agree 

2.1. What is the girl's name? 




2.2. What are they talking about? 





59 























The EUROCALL Review, Volume 25, No. 1, March 2017 


2.3. What is the girl's telephone 
number? 




2.4. What feelings may the boy 
have for the giri? 





3. Are you in favour of having the questions embedded in the image (in addition to 
having them on paper) in the audiovisual comprehension test of the final exam? 


a) No 

b) I am not sure 

c) Yes 

4. Next there is a space where you can comment on any aspect of the activity. You may 
reply in Spanish or Portuguese. 

Task 2. Fragment of the film Los peores ahos de nuestra vida. 

5. Based on your overall experience completing the task, to what extent do you agree 
or disagree that the subtitled questions have helped you? Mark your answer with a 
circle. 

a) Disagree 

b) Neither agree nor disagree 


c) Agree 

6. Individually, to what extent do you agree or disagree that subtitled questions have 
helped you? Complete the table with "X". 



Disagree 

Neither agree nor 
disagree 

Agree 

6.1. What is the redheaded boy talking about? 




6.2. What is the girl's name? 




6.3. What feelings may the red-haired boy have for 
the girl? 





7. Are you in favour of having the questions embedded in the image (in addition to 
having them on paper) in the audiovisual comprehension test of the final exam? 


a) No 

b) I am not sure 

c) Yes 

8. Next there is a space where you can comment on any aspect of the activity. You may 
reply in Spanish or Portuguese. 
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